Advances in automated speech recognition and natural language processing have made possible virtual personal assistants. These virtual assistants have found application in a plethora of everyday activities from helping people manage daily schedules and appointments, to searching the Internet for their favorite songs. However, being primarily speech driven, such virtual agents are inherently limited in their ability to sense and understand user behavior and thereby adequately address their needs. Human interaction is a highly complex interplay of verbal and non-verbal communication patterns that among other skills demonstrates a keen ability to convey meaning through finely calibrated facial expressions. Recent research in autonomous avatars aims to develop powerful human-computer interfaces that mimic such abilities. However, producing realistic facial expressions in avatars that are appropriately contextualized and responsive to the interacting human remains a significant challenge.